Fault-Tolerant vSphere Backup Automation

Headless Script Deployment & Resiliency Reference Guide

Target Environment: Mac Studio Native ARM64 Architecture

1. Architecture and Guardrail Mechanics

This automated implementation addresses infrastructure constraints by introducing strict telemetry metrics and defensive code architecture:

Crash Immunity: All high-risk operational steps (API calls, OVF streams, disk reads/writes, cloud connections) are encapsulated within isolation handlers. Any unexpected error triggers a Google Chat alert and skips the faulty VM to proceed with the remaining queue rather than terminating the global pipeline.
Disk Safety Verification: Before invoking an unthrottled storage export stream, the script evaluates the physical layout of the partition mount on the Mac Studio. It references the exact committed byte usage of the target VM via vSphere telemetry metrics. If the VM size exceeds available disk space, it prevents a local volume overflow condition, logs a warning, alerts your Google Chat workspace, and gracefully moves to the next machine.
Asynchronous Headless Lifespan: Designed to run detached via standard session processes (nohup), eliminating any dependency on your open SSH window.

2. Complete Resilient Automation Source Code

Save this script exactly as backup_resilient.py on your Mac Studio. It uses native Python standard libraries and requires no extra software installations.

backup_resilient.py

import os
import json
import subprocess
import sys
import shutil
import re
import urllib.request
import traceback
from pathlib import Path

# Paths Setup
HOME = str(Path.home())
STATE_FILE = os.path.join(HOME, ".vsphere-backup/state.json")
EXPORT_DIR = os.path.join(HOME, "Downloads/vsphere_scratch")
WEBHOOK_URL = "https://chat.googleapis.com/v1/spaces/AAQAL-_F754/messages?key=AIzaSyDdI0hCZtE6vySjMm-WEfRq3CPzqKqqsHI&token=lxpCnaETOeOv7OYKcsE71uaapaoFBb9qb0PBww-t9_g"

os.makedirs(os.path.dirname(STATE_FILE), exist_ok=True)
os.makedirs(EXPORT_DIR, exist_ok=True)

def send_chat_notification(text):
    \"\"\"Sends a secure, formatted payload to the Google Chat Webhook.\"\"\"
    try:
        payload = json.dumps({"text": text}).encode('utf-8')
        req = urllib.request.Request(
            WEBHOOK_URL,
            data=payload,
            headers={'Content-Type': 'application/json; charset=UTF-8'}
        )
        with urllib.request.urlopen(req) as response:
            return response.read()
    except Exception as e:
        print(f"Failed to transmit Google Chat alert: {e}")

def run_command(command, env_updates=None):
    \"\"\"Executes shell sub-processes securely and captures runtime output streams.\"\"\"
    current_env = os.environ.copy()
    if env_updates:
        current_env.update(env_updates)
    result = subprocess.run(command, check=True, capture_output=True, text=True, env=current_env)
    return result.stdout.strip()

def load_state():
    if os.path.exists(STATE_FILE):
        with open(STATE_FILE, 'r') as f:
            return json.load(f)
    return {}

def save_state(state):
    with open(STATE_FILE, 'w') as f:
        json.dump(state, f, indent=4)

def parse_vm_committed_gb(vm_path):
    \"\"\"Queries vSphere to calculate the true committed physical gigabytes of a VM.\"\"\"
    try:
        raw_info = run_command(["govc", "vm.info", "-r", vm_path])
        match = re.search(r"Storage committed:\s+([0-9.]+)(GB|MB|TB)", raw_info)
        if match:
            value = float(match.group(1))
            unit = match.group(2).upper()
            if unit == "MB":
                return value / 1024.0
            if unit == "TB":
                return value * 1024.0
            return value
    except Exception as e:
        print(f"Size parsing exception for {vm_path}: {e}")
    return None

def main():
    required_envs = ["GOVC_URL", "GOVC_USERNAME", "GOVC_PASSWORD", "GOVC_INSECURE"]
    for env in required_envs:
        if env not in os.environ:
            print(f"Critical Configuration Error: Environment variable {env} not set.")
            sys.exit(1)

    state = load_state()
    
    try:
        print("Polling inventory topology...")
        raw_vms = run_command(["govc", "find", ".", "-type", "m", "-runtime.powerState", "poweredOff"])
        all_vms = [line for line in raw_vms.split('\\n') if line]
    except Exception as e:
        error_msg = f"❌ *Critical Pipeline Failure*: Failed to communicate with vSphere API.\\n`{e}`"
        send_chat_notification(error_msg)
        sys.exit(1)

    if not all_vms:
        print("No powered-off target nodes discovered.")
        return

    pending_vms = [vm for vm in all_vms if state.get(vm.split('/')[-1]) != "COMPLETED"]
    total_to_process = len(pending_vms)

    if total_to_process == 0:
        print("All target resources match valid cloud signatures. Sync complete.")
        return

    send_chat_notification(f"🚀 *Resilient Migration Pipeline Initiated*\\nQueue contains *{total_to_process}* virtual machines scheduled for migration.")

    for idx, vm_path in enumerate(pending_vms, start=1):
        vm_name = vm_path.split('/')[-1]
        remaining_count = total_to_process - idx
        ova_dir = os.path.join(EXPORT_DIR, vm_name)
        
        print(f"\\n--- Processing Machine [{idx}/{total_to_process}]: {vm_name} ---")
        
        # Comprehensive try/except block guarantees that an error in one VM won't crash the loop
        try:
            # 1. Pre-flight Disk Space Safety Check
            vm_size_gb = parse_vm_committed_gb(vm_path)
            free_space_bytes = shutil.disk_usage(EXPORT_DIR).free
            free_space_gb = free_space_bytes / (1024**3)
            
            print(f"[{vm_name}] Telemetry Analysis -> Required Space: {f'{vm_size_gb:.2f} GB' if vm_size_gb else 'Unknown'} | Mac Studio Available: {free_space_gb:.2f} GB")
            
            if vm_size_gb and (vm_size_gb >= (free_space_gb - 5.0)):  # 5 GB safety buffer margin
                space_alert = (
                    f"⚠️ *Skipped Oversized Target*: `{vm_name}`\\n"
                    f"❗ *Reason*: Required space ({vm_size_gb:.2f} GB) exceeds available Mac Studio storage ({free_space_gb:.2f} GB).\\n"
                    f"📋 Queue Status: `{remaining_count}` machines left in active queue."
                )
                print(f"CRITICAL STORAGE OVERFLOW PREVENTED: Skipping {vm_name}")
                send_chat_notification(space_alert)
                continue

            # 2. Optimized OVF Stream Extraction
            if state.get(vm_name) != "DOWNLOADED":
                print(f"[{vm_name}] Initializing unthrottled HTTP/1.1 file export...")
                os.makedirs(ova_dir, exist_ok=True)
                
                run_command(
                    ["govc", "export.ovf", "-vm", vm_path, ova_dir],
                    env_updates={"GODEBUG": "http2client=0"}
                )
                state[vm_name] = "DOWNLOADED"
                save_state(state)

            # 3. Stream Upload to Cloud & Purge Local Blocks
            print(f"[{vm_name}] Transporting payload blocks to Google Drive remote partition...")
            run_command([
                "rclone", "move",
                ova_dir,
                f"gdrive:vsphere_backup/{vm_name}",
                "--drive-chunk-size", "64M",
                "--retries", "5"
            ])
            
            # Finalize State Verification
            state[vm_name] = "COMPLETED"
            save_state(state)
            
            # Post success alert
            success_msg = (
                f"✅ *Migration Completed*: `{vm_name}`\\n"
                f"📊 Progress Vector: Finished [{idx}/{total_to_process}] — *{remaining_count} VMs left*."
            )
            send_chat_notification(success_msg)
            print(f"[{vm_name}] Execution pipeline loop completed successfully.")

        except Exception as e:
            # Catch anomalies, keep execution thread alive, and alert user via Google Chat
            tb_string = traceback.format_exc().split('\\n')[-2]
            runtime_fault_alert = (
                f"💥 *Execution Exception Intercepted*\\n"
                f"❌ *Machine Node*: `{vm_name}`\\n"
                f"🚨 *Error Dump*: `{e}`\\n"
                f"📍 *Location*: `{tb_string}`\\n"
                f"⏳ *Action*: Skipping node safely; automation queue shifting to next machine."
            )
            print(f"EXCEPTION BLOCKED FOR VM {vm_name}: {e}")
            send_chat_notification(runtime_fault_alert)
            continue

    send_chat_notification("🏁 *Automation Execution Run Finished* — All safe assets processed.")

if __name__ == "__main__":
    main()

3. Execution and Deployment Manual

Follow these steps to deploy the script securely inside a detached session loop:

Step A: Establish Local Session Parameters

Export your active vSphere variables inside your current shell workspace on the Mac Studio:

Session Authentication Setup

export GOVC_URL="10.11.11.13"
export GOVC_USERNAME="Administrator@VSPHERE.LOCAL"
export GOVC_PASSWORD='TDN1hzg7xth0fmh!xwh'
export GOVC_INSECURE="true"

Step B: Start Background Session

Invoke the pipeline script behind a standard hangup protection layer (nohup). This detaches the job entirely from your user profile environment:

Headless Background Execution

nohup python3 ~/vsphere-backup/backup_resilient.py > ~/vsphere-backup/run_output.log 2>&1 &

Safe Disconnection Sequence

Once you execute the line above, you are free to type exit or terminate your MacBook Pro terminal app. The Mac Studio processor thread will run continuously in the background.

Step C: Raw Telemetry Stream Tracking

If you want to view local file growth or manual operations output while the automated job processes, reconnect to your Mac Studio shell and monitor the live log file:

Log Monitoring Output

tail -f ~/vsphere-backup/run_output.log

4. Resumption and Recovery Guide

If a total network blackout drops connection to both vSphere and Google Drive simultaneously, recovery is simple. You don't have to clean anything up manually:

Kill any stalled threads on the host Mac Studio: pkill -f backup_resilient.py
Re-run the deployment commands shown in **Step A** and **Step B** above.
The script reads ~/.vsphere-backup/state.json. It bypasses completed uploads, detects if an export file was only partially downloaded, and automatically handles cleaning up and resuming your progress.